Comprehensions, Maps, Lambdas, Generators and Decorators

The following are frequently used in Python and a good understanding of each is required.

Comprehensions

Comprehensions in Python provide a concise method of creating a new data structure from an existing one whilst filtering and or modifying the data.

List comprehensions

A list as we know is a sequence of data values akin to arrays in other languages. We know how to iterate over lists using 'For' and 'While' loops, let's look at how we use a comprehension to iterate over a list.

Prime Numbers Take a list of numbers from 2 to 100 and create a list of all the prime numbers.

For those who have forgotten all about prime numbers here is a quick recap.

A prime number is a number, excluding 1, that can be divided only by itself or 1. For example, 2 can be divided by itself or 1 and is therefore a prime number. 6 on the other hand can be divided by itself, 1, 2 and 3, therefore it is not a prime number.

Before we look at the list comprehension for prime numbers, let's look at a standard method using 'for loops'.

for x in range(2, 101):
    for y in range(2, x):
        if x % y == 0:
            break
    else:
        print(x, end=" ")

Here we use two for loops, one nested inside the other. The second with a simple conditional expression using the % modulus operator. Each number between 2 and 101 is divided by all numbers up until itself until the modulus is 0. If it is never 0 then the number is a prime number.

One other interesting thing about the above for loop is the use of the else under the for and not the if. Normally we assoicate an else with and if statement, but in Python you can use it to execute code after the loop has expired. In this case it will obly get executed if the break statement in the if statement is not applied. The break statement will exit the loop completely.

Run that and note the prime numbers.

Now let's translate that to a list comprehension.

primes = [x for x in range(2, 101) if all(x % y != 0 for y in range(2, x))]
print(primes)

The above comprehension is almost the same except that it is contained within [] to indicate that the result should be a list called primes. Each x that is a prime number will be placed in the list primes. The ordering of the loop constructs is also different with the use of if with an all operator to indicate that we want to test each number in the range 2 to x represented as y.

Dictionary comprehension

Pretty much the same as list comprehensions but dealing with dictionary key-value pairs instead of indexed items. Obviously the result is a dictionary.

new_dict = {k:v for k,v, in {1: 'a', 2: 'b', 3: 'c', 4:'d'}.items() if v < 'c'}
print(new_dict)

Tuple comprehensions??

There is no tuple comprehension in Python, but you can get the results of a list or dictionary comprehension into a tuple by wrapping a list comprehension in a tuple function. If you did the same for a dictionary comprehension you would just get the relevant keys. Try it out.

primes = tuple([x for x in range(2, 101) if all(x % y != 0 for y in range(2, x))])
print(primes) # Prints a tuple containing the prime numbers

Maps

Python maps are generally used to iterate over data structures applying functions or expressions to the items in the data structure. You can think of it as mapping expressions over data.

Let's look at an example

def square(x):
  return x ** 2

a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b_list = list(map(square, a_list))
print(b_list)

What's happening here is the map function has two parameters square, the name of the function to call and a_list the list of numbers. For each number in the list a_list, the map function calls the function square and passes the number as a parameter. The square function does its work and returns the result.

Note calling a function without a return value is not going to work as the result of the mapping will be lost.

The results of the mapping function above are placed in a list, as can be seen we wrap the map() function call in a list() function which ends up in the variable b_list

The following example will convert strings to uppercase.

def upper(x):
  return x.upper()

a_list = ["richard", "tayfun", "pablo"]
b_list = list(map(upper, a_list))
print(b_list)

Lambdas

Python Lambda functions are often called anonymous functions. Unlike normal functions, Lambdas are shorthand, mostly one off usage functions in the sense that they are not required as a reusable function that is callable from multiple locations in the code. A typical use is a one of calculation for a map function.

Let's take our square root example above and make it use a lambda function instead of a normal function.

a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b_list = (list(map(lambda x: x ** 2, a_list)))
print(b_list)

Filtering a list

a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b_list = (list(filter(lambda x: x > 5, a_list)))
print(b_list)

Celsius to Fahrenheit conversion))

a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b_list = (list(map(lambda x: (x * 1.8000) + 32, a_list)))
print(b_list)

Now we have a one liner map function with a lambda function and a list as the two components.

You can also return a lambda function from a normal function and call it with a value.

def multiply (n):
  return lambda x: x*n

by_five = multiply(5)
by_ten = multiply(10)
by_fifteen = multiply(15)

print(by_five(5))
print(by_ten(5))
print(by_fifteen(5))

Generators

Python generators are instances of iterator functions that use a yield statement to return the code event control back to the calling code. Hence, any function that has a yield statement in it can be considered a generator. To yield means to give back control, in this case to the code that called the generator. As with all iterators you can invoke the next function to continue with its execution.

Generators have numerous usages, not least walking through large sets of data asynchronously. Imagine you want all the records in a database that have a certain key in them, but you want to process those records as they arrive rather than collect them all first. Using a generator can facilitate that.

Here we shall stick with a few introductory examples to get you familiar with the concepts.

Recalling the prime number loops and comprehension earlier, i.e. the following example

for x in range(2, 101):
    for y in range(2, x):

        if x % y == 0:
            break
    else:
        print(x, end=" ")

In the for loop above we continue the loop until all prime numbers from 2 to 100 have been found. Let's turn this into a generator function

def prime_generator():
    for x in range(2, 101):
        for y in range(2, x):
            if x % y == 0:
                break
        else:
            yield x

for n in prime_generator():
    print(n, end=" ")
    print("times 2 = ", n*2)

The yield in the prime_generator function yields the variable x and returns to the calling loop which does some work, in this case a couple of print statements.

We can use the generator with the next function thus,

def prime_generator():
    for x in range(2, 101):
        for y in range(2, x):
            if x % y == 0:
                break
        else:
            yield x

primes = prime_generator()
print(next(primes))
print(next(primes))
print(next(primes))

One last example takes us back to the fact that in Python Tuples do not have comprehensions. However, it is possible to generate a generator object from what looks like a tuple comprehension.

# Creates a generator
primes = (x for x in range(2, 101) if all(x % y != 0 for y in range(2, x)))

# Iterate over the generator using the next function.
print(next(primes))
print(next(primes))
print(next(primes))
print(next(primes))
print(next(primes))
print(next(primes))

Decorators

Python decorators are basically pointers to other functions that you wish to run before proceeding with some other function call. A Python decorator name is preceded by an @ symbol as in @my_decorator. Decorator functions generally have inner functions that do the work of the decorator.

Decorators are commonly used to apply criteria for using functions, such as login control, access permissions, filters, conversions or expressions on data etc. etc.

We'll take a look at a couple of uses below, the first of which will use our prime number generator as an example.

def generate_random(func):
    import random

    def get_randoms():
        """
            Inner decorator function that does the heavy lifting
        """
        x = random.randrange(10)
        y = random.randrange(10, 102)
        return func(x, y)

    return get_randoms

@generate_random
def prime_numbers(r1, r2):
    print(f"generating primes from {r1} to {r2}")
    return (x for x in range(r1, r2) if all(x % y != 0 for y in range(2, x)))

primes = prime_numbers()

for i in primes:
    print(i)

We have a function prime_numbers that is decorated with a decorator called generate_random, which is in itself a function that generates two random numbers and passes those back to an instance of the prime_numbers function.

The decorator function has two parts, the main decorator function generate_random, which has a parameter func, a reference to the function that it is decorating, i.e. prime_numbers, and an inner function, get_randoms. This is the core part of the decorator that generates the two random numbers.

The inner function is called using return get_randoms from the main decorator function, and returns an instance of the func function reference with the two random numbers, which the main decorator function returns as a consequence of the return get_randoms statement.

Notice the lack of () brackets in the call to get_randoms.

Our next example represents a cut down version of a more fruitful use of decorators, that of access role control.

def permissions(required_access_role):

    def is_accessible(func):
        def wrapper(user_access_role):
            if required_access_role == user_access_role:
                return func(user_access_role)
            else:
                return False

        return wrapper

    return is_accessible

@permissions("admin")
def do_admin_work(access_role):
    print("Doing very important admin work")

do_admin_work("basic")

This is a tad more complicated than the first decorator, instead of a single inner function it has a second inner function inside the first inner function.

The reason for this is we want to use the parameters from the decorated function do_admin_work, explicitly the access_role parameter. Using this second function allows us that access. As can be seen we have a parameter access_role for the wrapper function.

This decorator also has its own parameter required_access_role which is set to admin. What this is telling us is that to access the do_admin_work function you need to have an access_role that is admin.

The decorator compares the required_access_role sent as a parameter in the actual decorator call @permissions("admin") and the user_access_role that is a parameter of the do_admin_work function. If they match it return the function which runs. If they don't match it returns false and the function does not get called. Normally you would handle the mismatch with some form of exception handling, but we haven't covered this subject yet.

That's it for this section, you can move on to the next part of this tutorial.

Comprehensions, Maps, Lambdas, Generators and Decorators

List comprehensions​

Dictionary comprehension​

Tuple comprehensions??​

List comprehensions

Dictionary comprehension

Tuple comprehensions??